Augmentation of Sets of Image Recordings 
Fiplri of the Invention 

5 The present invention relates to augmenting sets of image recordings. As used herein, 
image recordings includes single photographs, sequences of photographs, and video 
recordings, all whether chemical or digital. 

Background of the Invention 

1 0 Classification of photographs, particularly those taken by individuals and famiUes, has long 
been a problem due to the effort involved in maintaining any organisation ofthe photos. 
What is needed is a logical organisation, valid over a lifetime, that requires minimal effort 
to implement and is intuitive to use when retrieving photos. 

15 It has been previously proposed to associate location (and orientation) data with digital 
photos. The location data can be derived in any suitable manner such as from a GPS system 
or by using information obtained from a cellular radio system. Thus, JBM Technical 
Disclosure 413126 teaches a digital camera provided with a GPS receiver. US 5,712,679 
discloses a locatable portable electronic camera which is arranged to send back image and 

20 location data when triggered, the location data being displayed on a map and the image 
being shown separately. Also of interest is US 5,389,934 which describes a portable 
locating system with a GPS unit that is operative to store a travel history of locations 
visited. 

25 Other proposals go further and disclose the management digital photographs by using an 
electronic map to show a thumbnail of each photograph at a corresponding position ofthe 
electronic map. Figure 1 ofthe accompanying drawings illustrates the main elements for 
implementing such an system, these element being a digital camera 3 equipped with a GPS 
receiver for determining camera location using signals from satellites 2, a PC 5 for 

30 receiving digital photographs 4 downloaded from the camera 3 together with GPS-derived 
location information about where each photograph was taken, an album program 6 for 
managing the downloaded photographs, a store 7 for storing the digital photographs (plus 
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location information), and a store 8 for storing map data (the stores 7 and 8 will generally 
be internal to the PC 5 but may by external). Such an arfangement is described; for 
example, in JP 10233985 A. 

5 The combination of location-tagged digital photographs withmap-based digital photograph 
albums should greatly facilitate the management of photographs for the ordinary user. 
However, it is important that the photograph management systems provided with the map- 
based electronic albums are also flexible and easy to use. In this respect the above- 
mentioned JP 1 0233985 A describes the possibility of the user classifying each photograph 

1 0 whilst JP 8335034A discloses the use of an icon to represent groups of photographs on a 
map display. 

It is an object of one aspect of the present invention to further facilitate the photographic 
experience by enabling a user to complete a set of photographs (and other image 
15 recordings). 

Certain arrangements described hereinafter utilize data bearer services and location 
services of cellular radio networks. Such networks are widespread though the afore- 
mentioned services are only now being taken up significantly. To facihtate an 
20 understanding of the following description, a brief review of cellular network technology 
and how the foregoing services can be implemented is given in the Annex appearing at the 
end of this description, reference being had to Figure 2 of the accompanying drawing that 
depicts a mobile location-aware service being provided to a mobile entity 20 via a Public 
Land Mobile Network (PLMN) 10 . 

25 



Summary of the Invention 

According to the present invention, there is provided a method of augmenting a set of 
image recordings, wherein a set of image recordings are taken and corresponding taken- 
30 image-recording location data is recorded to indicate the locations where the image 
recordings were taken; and wherein desired-image-recording location data is also recorded 
to indicate at least one location for which the user desires an, or a fiuther, image recording; 





3 

the desired-image-recording location data being subsequently used to retrieve one or more 
corresponding image recordings. 

Other aspects of the invention are set forth in the accompanying claims 

5 

Brief Description of the Drawings 

Embodiments of the present invention will now be described, by way of non-limiting 
example, with reference to the accompanying diagrammatic drawings, in which: 
• Figure 1 is a diagram of a known map-based photographic album system; 

10 .Figure 2 is a diagram of a known communications infrastructure usable for 



• Figure 3 



. Figure 4 



transferring voice and data to/from a mobile entity; 

is a diagram of an electronic photographic album system showing the 

five operating modes of an album program; 

shows the fields of a photo record and group record of the Figure 3 
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30 . Figure 12 



. Figure 5 
. Figure 6 



. Figure 7 



. Figure 8 



. Figure 9 



. Figure 10 
. Figure 11 



system; 

shows state data items maintained by the album program of Figure 3; 
shows a typical display output during a "Catalogue" operating mode of 
the Figure 3 system; 

shows a typical display output during a "Map View" operating mode of 
the Figure 3 system; 

shows a typical display output during a "Photo Show" operating mode of 
the Figure 3 system; 

is a diagram illustrating the transfer of location data fi-om a cell phone to 
a digital camera; 

is a state diagram for the location-data transfer process of Figure 6; 
is a diagram illustrating the transfer of camera image data via a cell 
phone and a PLMN to a PC running the Figure 3 album program, the 
image data being location stamped with the cell phone location during 
transfer; 

is a diagram showing the independent transfer of image data and 
location data to a PC running the Figure 3 album program; 



. Figure 13 is a state diagram of a location log function of a mobile entity equipped 

with location discovery means; 
. Figure 14 is a diagram illustrating the main steps of the Load and Catalogue 

operating modes of the Figure 3 album program in the case of image 
5 data and location data being separately provided; 

. Figure 15 is a diagram illustrating the matching of location data to images by 

matching patterns of timestamps; 
. Figures 16A-D show a user-effected correction of mismatched sequences of images and 

location data; and 

10 . Figure 17 is a diagram showing the recording of the location of desired but not 

taken photos and the subsequent retrieval of matching images. 



Rest Mode of Carrying Out the Invention 

15 Figure 3 depicts a photo system in which a digital camera 3 provided with location 
determining means (such as a GPS receiver) is used to generate digital photos 4, each 
photo (also referred to as 'image data') 4 being stamped with location data indicating where 
the photo was taken. Other data may also be associated with each photo, such as a 
timestamp, a camera ID, and a user ID; such associated data (including the location data) is 

20 herein referred to as photo meta data. The photos and their meta data are downloaded by 
any suitable means (USB connection, removable storage device, etc) into a PC 5 where an 
album program 50 serves to store the photos in a photo store 7 and the photo meta data in a 
meta-data database 9 (each photo and its meta data being linked by a suitable key 
associated with both). The album program also has access to a map store 8. The stores 7 

25 and 8 and the meta-data database can be on the PC or external. 

The album program enables users to catalogue, manage and view their photos through a 
map-based interface, the photos being represented on a displayed map by a marker 
indicating the location they were taken. 



30 



More particularly, the album program comprises five main operating modes 5 1 to 55 and a 
user can move between these modes or quit the program by making an appropriate choice 
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(for example, by soft keys displayed on the PC display). Figure 3 indicates for each mode 
the main choices available to the user (for example, the label -View" in Start Up Mode 
block 5 1 indicates that the user can choose to change to the View Map Mode 54. The role 
of each operating mode is as follows: 
5 Start Up Mode 5 1 - This is the initial mode upon start up of the album program and it 

permits a user to select either the Load Mode or the Map View 

Mode, or to quit the program. 
Load Mode 52 - In this mode, the user can download data from camera 3 ; when the 

user has finished, he/she indicates this (see the "Done" label) and 
10 the mode changes either to the Catalogue Mode 53 if any photos 

have been loaded, or back to the Start Up Mode 51 if no photos 

were loaded. 

Catalogue Mode 53 - In this mode, the user can manage newly loaded photos using a map 
based display, this management including assigning them to one or 
1 5 more groups (sets of related photos). From this mode, the user can 

move back to the Load mode to load more photos or to the View 
Map mode for browsing the photo album; the user may also choose 
to quit the program. 

Map View Mode - The Map View Mode 54 is the mode in which a user can browse the 
20 album and select photos for viewing. Browsing is on the basis of 

displaying maps with the location of photos indicated. From the 
Map View 54, a user can move to the Load, Photo Show, or 
Catalogue Modes or quit the program. 
Photo Show Mode - hi this mode, the user can view a photo selected in the Map View 
25 Mode; the user can also step through a series of related photos. 

From the Photo Show Mode, the user returns to the Map View 
Mode. 

Figure 4 shows the meta data record 56 held in database 9 for each photo, it being 
30 appreciated that some of the fields may be empty for any particular photo. The fields 
comprise: 

Album ID - This is a unique identifier for the album. 



Camera ID 



User ID 



Photo ID 
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Accession Date 

Location Data 
15 Date/Time Taken 
Short Title 
Description 
Semantic Loc. 

20 



This is a camera identifier that may be either supplied automatically 
by the camera in the photo meta data or added by the user when 
downloading photos. 

This is a user ID which again may be either supplied automatically 
by the camera in the photo meta data or added by the user when 
downloading photos 

This is a unique photo ID provided by the album program and can 
conveniently be made up of a load batch number (a new batch 
number being assigned for each session of downloading data firom a 
camera) and a number-in-batch identifying the photo fi-om others in 
the same batch. 

This is the data of loading of the photo by the album program 
(photos in the same batch will have the same accession date). 
The location data provided with the photo by camera 3. 
The timestamp data provided with the photo by camera 3. 
A short descriptor of the photo provided by the user. 
A fuller user-provided description of the photo. 
. A user-meaningful location description (e.g. Eiffel Tower) as 
opposed to the coordinates provided by the location data. This field 
overlaps in intent with the two preceding fields and is optional. 



With respect to user ID, where this is supplied automatically by the camera, the user ID 
will have been set into the camera at some stage by the user. The camera can be provided 
with suitable means for enabling each of several users to set in their ID at every usage 
25 and/or the means for enabling several different users to set in and store their IDs with each 
such user selecting their ID from the stored IDs each time that user starts to use the camera; 
input ofID data can conveniently be donebytransferfromacomputer thereby avoidingthe 

need for an input keypad associated with the camera. Alternatively, the camera can be pre- 
programmed with a set list of identifiers (numbers, icons, colours, animal types, etc) and 
30 users choose which identifiers to employ to distinguish amongst them; in this case, the 
camera simply needs to be provided with input means for enabling a user to select their 
identifier from the programmed list of identifiers. Thus, a camera intended for family use 



may have pre-programmed animal icons as identifiers with the mother and father choosing, 
for example, icons of a dog and cat and their three children choosing lion, tiger and rabbit 
icons respectively. Of course, to handle cases where icon identifiers are used, the album 
program is preferably adapted to store and manage user IDs of this form. 
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Current Mode 
Previous Mode 



The database 9 also holds group records 57 each relating to a group of user associated 
photographs. Each group record comprises a group name, an indicator of a group icon to be 
used to represent the group, a brief description of the group, and a linked list of the IDs of 
1 0 photos making up the group. A photo can reside in non, one or more group. 

Figure 5 shows the main state data items 59 maintained by the album program so that it 
knows what operational point it has reached and how to restore itself to certain prior 
conditions as necessary. These data items comprise: 
The current operating mode. 

The previous operating mode, if any. This is required when about to 
enter the Map View mode from the Catalogue Mode - if the user is 
merely returning to the Map View Mode after having gone into the 
Catalogue Mode (for example, to change meta data about a photo), 
20 then the user will expect to reum to the same map and display as 

before whereas if the user is entering the Map View Mode from the 
Catalogue Mode after having catalogued a newly loaded set of 
photos, the user will have different expectations. 
Current Photo - This is the currently selected photo, if any. The selected photo may 
25 be one currently be displayed in fiill or merely represented by an 

icon or thumbnail. 

- The batch number of the current batch of photos or, where photos 
from more than one batch are being examined, then the batch 
number of any currently selected photo. 

30 Current Group - The currently selected group, if any. 

- Details of any filter being applied to the selection of photos in the 
Map View and Photo Show Modes. 



Current Batch 



Current Group 
Filter Details 
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Most recent: 

Map View map - The most recently viewed- Map View Mode map (including area 
displayed) 

Catalogue map - The most recently viewed Catalogue Mode map (including area 
displayed) 

Other features of the album program will be described below as part of the description of a 
typical sequence of operation. 



A userjwith photos to download to the album program starts the program and chooses the 
"Load" option presented by the Start Up Mode program code. The Load Mode interacts 
with the user to enable the downloading of photos and meta data from the camera 3, photo 
m data (Batch and number-in-batch) and accession date being automatically added by the 
program to the meta data of each photo. The user may also be enabled to add in user ID 
data for the whole batch, overriding any such data coming from the camera. Upon 
termination of loading, the user selects "Done" and the album program automatically 
progresses to the Catalogue Mode to enable the user to carry out cataloguing fiinctions in 
relation to the newly-loaded batch of photos. 

In the Catalogue Mode, the album program generates a display of the form shown in Figure 
6 comprising a central map area 61, left and right margin areas 62A, 62B and upper and 
lower control bars. The map displayed in map area 61 is sufficient to encompass the 
locations registered for the newly loaded batch of photos. A thumbnail 63 of each new 
photo is shown in one or other of the margin areas 62 A,B and a lead line 65 connects each 
thumbnail 63 to a corresponding marker 64 showing on the map the location where the 
photo was taken. The use of margins to show the thumbnails and lead lines to indicate the 
corresponding map locations is preferred as being less cluttered than trying to place the 
thumbnails directly in the correct locations on the map. 

The upper control bar comprises three controls 66, 67, 68 that provide access to the 
following functionality, generally in respect of a pre-selected photo (this pre-selection 
being effected by clicking on the corresponding thumbnail): 
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Show Photo Control 66 - this displays the photo corresponding to a selected thumbnail 63, 
with return to the Catalogue map being effected by a mouse chck; 
Edit Photo Details Control 67 - this displays the record details 56 of a selected thumbnail 
and enables editing of these details; 
5 Group Control 68 - this control permits groups to be created, and aphoto to be assigned to 
one or more groups (the photo ID being added into the group record 57). 

The group control comprises a drop-down box 68A operated by control element 68B, the 
box normally displaying a currently selected group, if any. The user may cause the box to 

10 drop down (by clicking on control element 68B) to show a list of available groups from 
which the user can select one, this list also including an option to create a new group. 
Selecting this latter option takes the user to a creation screen where the user enters details 
of the new group. The details of a currently selected group can also be brought up for 
editing by operating (clicking on) control element 68C. To assign a photo to the current 

1 5 group, the corresponding thumbnail is selected and then the "Apply" control element 68D 
is operated. Preferably, double clicking the "Apply" control gives the mouse cursor the 
"Apply" power so that any thumbnail selected using the cursor is assigned to the current 
group (clicking on the Apply element again removes this power). 

20 By way of example, a user may decide to create a group for photos taken on a particular 
holiday or a group for all photos related to a current or past abode. The group icon can be 
selected from a set of available icons or created by the user. Thus a current-abode group 
may have a house icon as its group icon whilst a holiday in London may be represented by 
a Tower Bridge icon. 

25 

The lower confrol bar includes a zoom confrol 69 that enables a user to zoom in or out 
around a particular point on the displayed map. More particularly, to zoom in on a target 
point, the "+" element of confrol 69 is selected, the display cursor placed on the target point 
and clicked. The "+" and "-" elements effect stepped, progressive, zooming; in confrast, 
30 the "Full In" element goes straight to closest zoom on a target point whilst "Full Out" 
returns to the original map display encompassing all of the newly loaded photos. 
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Controls 70, 71 and 72 respectively enable a user to change to the Load Mode, change to 
the Map View Mode, and Quit the program. 

The album program is preferably operative to accept photos for which there is no meta 
5 data, including no location data. In this case, the corresponding meta data record initially 
only contains the album-generated data (Photo ID, accession data), and the Catalogue 
Mode is arranged to represent these photos but without lead line or location marker until 
such a time as the user enters location data into the location field of the photo record 56 
either directly or by a facility for adding this data by pointing to a location on the map 
10 display. 



Once a user has finished editing the photo meta data and assigning the photos to groups, 
the user may decide to browse the album and accordingly operates the "View" control 71 . 
1 5 Figure 7 depicts a typical Map View Mode display; for ease of understanding, the same 
references have been used on corresponding elements appearing in the Catalogue Mode 
and Map View Mode. 

The starting map displayed in the Map View Mode is, for example, a world map or a map 
20 encompassing the locations of all the photos recorded in the album; alternatively, the map 
displayed could be the same map as being used in the Catalogue Mode before transiting to 
the Map View Mode. Initially, all photos relevant to the starting map will be represented 
either as thumbnails 81, individual photo icons 80, group icons (see current-abode group 
icon 85), or concentration icons (see icon 82). A concentration icon represents a number of 
25 photos that are not in a common group but were taken in the same general area and cannot 
be individually represented at the current map display resolution; the area concerned can be 
represented by a bounding circle 83. Where a concentration icon only encompasses photos 
that belong to a common group, the concentration icon is replaced by the group icon. 
Similarly, where a concentration icon encompasses at least a threshold number (e.g. 5)of 
30 photos that belong to a common group but other photos as well, then the group icon is 
shown alongside the concentration icon. If the threshold is crossed for several groups then 
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each group icon will be shown (in determining whether the threshold is crossed, if a photo 
belongs to more than one group, it is counted towards the threshold for each group). 

Zooming in and out is controlled by the same control 69 as already described for the 
5 catalogue Mode. If zooming in permits the photos of a concentration to be represented 
individually then this is done, the photos being collapsed back to a concentration icon 
and/or group icon on zooming back out. Scrolling the map display left, right, up or down is 
effected by scroll control 75 (or else scroll bars can be used). 

1 0 To minimise clutter, a user may set an option (through a "customise menu", not shown) by 
which all photos of a group are initially represented by the corresponding group icon even 
if there is room to display the thumbnails of the group photos encompassed by the currently 
displayed map. In this case, the group icon is displayed with its location marker at the 
location of a user-specified one of the photos of the group (to implement this, a fiirther 
1 5 group-control element "set leader" can be provided which, when activated, would store the 
photo ID of a currently-selected photo into an extra field of the group record of the current 
group, the location of this photo being the "location" of the group). To give access to the 
individual photos of a group, one or both of the following mechanisms can be used: 

Single clicking on a group icon brings up a scrollable hst of the photos in the group 
20 (preferably with date taken information and short title). Mouse rollover of a photo 

entry in the hst causes the location marker of that photo to be displayed on the Map 
View (the photo's thumbnail can either be shown in the scrollable list all the time the 
list is displayed, or adjacent its location marker but only whilst the latter is 
displayed); moving the mouse cursor off the entry causes the location marker to 
25 disappear. Clicking the list entry causes the thumbnail to be inserted into a margin 

area with a lead line to a corresponding location marker on the map, this thixmbnail 
being retained following closure of the group photo list. A variant of the above is to 
have display of the list accompanied by display of location markers for all the group 
photos encompassed by the current map - rollover of a list entry would then 
30 highlight the relevant location marker. 

double clicking on a group icon unpacks the group and causes its photos to be 
represented as individual photos (or in concentrations). 
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Generally, whenever a particular ph5to is selected (for exaniple, by clicking on it) the name 
of the related group (if any) is displayed in the group control element 68 A - if a photo 
belongs to more than one group, these groups are viewable by opening up the dropdown 
5 box using control 68B. The details of the selected photo can then be viewed (but not 
edited) by operating control 78 or the photo viewed by operating Show Photo control 74 
which causes the mode to switch to the Photo Show Mode. The details of the current 
group, if any, can be viewed by operating the view group control 68E. 

10 Filter control 76 enables a user to select which photos are to be represented on the map 
display. Selection criteria can, for example, include one or more of the following: 
Date range of when the photo was taken; 

- User ID 
Camera ID 

15 - Group name 

- Batch ID 
Accession date 

Key word in short title / description / semantic location. 
Access to a particular's user's photos can be password protected 

20 

If the user wishes to edit the details of a photo or group, the user must select the relevant 
photo and return to the Catalogue Mode; the map and photos displayed in the Catalogue 
will be those of the same batch as the selected photo. 

25 Figure 8 shows the Photo View Mode display brought up by clicking the Show Photo 
control 74 in the Map View Mode when a photo is selected. A full size image 79 of the 
photo is displayed and the user can view the photo and group details using the controls 67 
and 68 respectively. Furthermore, a control 77 permits the user to view related photos in 
the same group (if photo is in more than one group, this will be the group appearing at the 

30 top of the dropdown box 68 A, a different group being selectable by dropping down the 
group Ust); the group photos are accessed, for example, in date/time of taking order. If a 
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photo is not associated with a group, then the album program permits photos of the same 
batch to be viewed, ordered by number. 

The provision of suitable program code for implementing the above-described event-driven 

5 functionality is within the competence of persons skilled in the art. 

A similar map-based album to that described above can also be used to classify and access 
other types of recording such as sound recordings, video recordings etc. Where the data is 
non-visual, the thumbnails and full-sized photo image representations of the above- 
described electronic photo album will be replaced by the corresponding representations for 

10 the recording concerned. 

TTnitm p T.ncation and Recording D ata - at the time of generation 

Of course, the vast majority of current cameras are not provided with location determining 
means. Nevertheless the foregoing map-based album can still be built up provided the user 

1 5 can activate a location determining device whilst located at the place a recording is being / 
has been / is about to be, made, hi the near future, many location-determining devices 
(such as GPS devices) will be widely deployed; potentially more significantly, location 
services will become widely available to users of mobile phones (see the Annex to this 
specification which describes the mobile radio infi-astructure and the provision of location- 

20 based services using such an infi-astructure). 

Thus it will become relatively easy for someone taking a photo to find out their location 
using their mobile phone. However what is additionally needed is some way of uniting this 
location information with the photographs. 

25 

One way of doing this is illustrated in Figure 9 where a digital camera 90 is provided with 
a communications link to receive location data fi-om a mobile entity 20 (here shown as a 
mobile phone, by way of example). More particularly, camera 90 comprises optics 91, 
sensor array 92, image processing block 99, control block 93, memory 94 for storing photo 
30 image data 95, and a communications interface 96. Cell phone 20 comprises, as well as its 
radio subsystem 22, a data handling subsystem 23,and communications interface 97. 
Interfaces 96 and 97 are compatible, enabling the camera 90 and cell phone 20 to inter- 
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communicate; interfaces 96 and 97 are, for example, suitable for establishing an infrared 
or short-range radio link between the camefa anB ceirphorie. 

Cell phone 20 also includes location-discovery means 29 by which the cell phone can 
5 ascertain its location, this location discovery being effected when control 28 (hard or soft 
button) is operated by the user. The location discovery means is, for example, a program 
run by the data handling subsystem for requesting location information from a location 
server of the mobile radio infrastructure; however, the location discovery means could 
alternatively be a GPS system built into the cell phone. Whatever form the location 
10 discovery means takes, when button 28 is operated, location data 98 is generated and is 
available in the phone for transfer to the camera 3. 

The data handling subsystem runs a transfer program for transferring the location data over 
a link establish between the interfaces 96, 97. The control block 93 of the camera is 

1 5 operative to receive this location data and associate it with the last-taken photo. Figure 1 0 
shows a top-level state diagram of how this process is managed by association functionality 
of control block 93. Normally the association functionality resides in a state 100 in which it 
is ready to receive location data through interface 96; whilst in this state, the camera can be 
used to take photographs and the association functionality remains in state 101. However, 

20 upon location data being passed from cell phone, the association fimctionality transits to 
state 101 in which the camera is blocked from taking a photograph. In state 101, the 
association fimctionality of control block 93 receives the location data and associates it 
with the last taken photo. Once this is done (and it generally will happen very rapidly) the 
association functionality returns to state 100. 



25 



Of course, it would also be possible to have the taking of the photo by camera 90 trigger 
the location discovery by the cell phone followed by transfer to the camera. 

Another way of uniting a digital photo and location data is illustrated in Figure 11 and 
30 involves uploading the photo image data 95 through the cell phone (via a link established 
between camera 90 and cell phone 20 through interfaces 96 and 97), to a network store 43 
of a service system 40 (arrow 105 represents this transfer). The service system 40 resides 
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either in the mobile infrastructure or is accessible via the latter over a data-capable bearer 
service. En route to the store, or upon loading into the store, location information 98 on the 
mobile phone is requested and associated with the photo image data 95; in the first case, 
the location data is obtained by the cell phone and associated with the image data as the 
5 image data is being transferred to the store 43 ,whilst in the second case, a control function 
42 of the store is operative to request the location data 98 from location 41 immediately 
upon the image data being received by service system 40. Of course, this method will 
generally need to be effected for each photo immediately it is taken since otherwise the 
location of the cell phone may not correspond to the location where the photo was taken. 

10 

The foregoing methods of associating separately generated image and location data at 
around the time of generation can equally be applied to other types of recording. 

Uniting Location and Recording Data - subsequent to when generated 
15 In many cases, it will not be possible, for whatever reason, to link the camera 90 with a 
cell phone or other available location discovery means (such as a stand-alone GPS device). 
For these cases, a location log can be created for subsequent correlation with the photos 
being taken. More particularly, 

as the camera is used to take a number of photographs, the photographs as items are 
20 distinguished from each other by an implicit (e.g. sequence position) or explicit 

location-independent reference associated with each; 

in association with taking each of at least some of said photographs, a mobile device 
that is separate fi-om the camera and is capable of effecting or triggering location 
discovery of its position, is used to generate location data indicative of the location at 
25 which the photograph was taken, this location data being stored together with an 

index matching the reference associated with the corresponding photograph; 
subsequently, the location data is united with the corresponding photographs by a 
correlation process using said references and indexes. 

The mobile device is, for example, a cellular-radio-based mobile device (phone or e.g. a 
30 PDA with mobile radio capability) capable of effecting location discovery such as by 

requesting location data from a location server; the mobile device may take other forms 

such as a standalone GPS device. 
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References can simply be position-in-sequence of photographs (in which case the 
corresponding indexes are similar ordering data). Alternatively, the references can be time- 
stamps - in this case, the indexes could be timestamps also (or, again, ordering data since 
5 timestamps are also this). 

The photos can be traditional (chemical) snaps and the uniting is done by printing labels 
with the location data, these labels then being stuck on the back of the snaps (preferably 
this location data takes the form of a map showing the location where the photo was taken) 
10 - in this case, the labels are numbered to correspond to photo numbers. 

Preferably, however, the photos are digital (or digitised) and the uniting of the photos with 
the location information is done in a PC or other computing device as part of the album 
program. Processes for effecting this uniting will be described hereinafter. 

15 

With regard to how the location data is transferred to the same computing device as the 
photo image data, a number of possibilities exist and Figure 12 illustrates three such 
possibilities in the case where the mobile device is a cell phone 20. More particularly. 
Figure 12 shows a camera 3 and cell phone device 20 both possessed by the same user. The 

20 cell phone 20 communicates with mobile radio infrastructure 1 0. Whenever a user takes a 
photo he/she operates a button 28 of the cell phone to cause the cell phone to trigger a 
determination of its location either by itself or through location server 41 of the PLMN 10. 
A log of location data on each photo taken is built up. In due course the user transfers the 
image data 95 from the camera 3 to computer 5 running the album program 50. As regards 

25 transfer of the location log to the computer, the following three possibilities are illustrated: 

a) - location data for each photo is accumulated in a location log 100 stored in the cell 

phone and subsequently transferred (see arrow 111) directly to the computer 5 over 
a wire link, infrared link or short-range radio link . 

b) - location data for each photo is accumulated in a location log 100 stored in the cell 
30 phone and this log is subsequently transferred (arrow 109) via a data-capable bearer 

service of the PLMN 1 0 to a store 47 (in the PLMN or a connected network, such as 
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the Internet). The location log is later retrieved by computer 5 from store 47 (see 
arrow 110). 

c) - Operation of cell-phone button 28 sends a request (arrow 107) to a log-service 
controller 44 of a log server system 40 to obtain the location of cell phone from 
5 location server 41 and store it in a log 100 held in store 45 of the service system, the 

identity of the log to be used being included in the request. The log 100 is 
subsequently retrieved by computer 5 from store 45 (see arrow 108). 

The same processes as described above can being effected for other types of recordings, the 
10 location data being separately determined and subsequently re-united with the recording 
concerned. In the case of a sound recording done, for example, on a tape recorder, the 
location data could even be provided by a digital camera equipped with GPS. 

It may be noted that giving a mobile phone the ability to store a location log (either in the 
1 5 phone itself or in the mobile infrastructure or in a connected network) is itself a usefiil 
feature. Thus whilst many location-based services simply require a one-ofif provision of 
location data or continually monitor location, the ability for a user to selectively trigger 
location determination for storing the resultant data to a log has value in its own right - for 
example, a user may wish to store the location of places visited whilst out walking or, as 
20 described above may want to log the locations of photos taken. Since the user may also 
want to use other location-based services at the same time, the user must be able to select 
when location information is to be logged. Further, since the user may want to log location 
information about different topics, the mobile phone (or other location-determination- 
triggering device) preferably permits a user to set up more than one log at a time and to 
25 select to which log a particular item of location data is to be stored. 

Figure 13 shows a controlling state machine for a location-log application capable of 
managing multiple location logs, the application running, for example, on a data handling 
subsystem of a mobile entity (such as a cell phone) that has means for discovering its 
30 location either directly or from a location server. Selection of the application sets the 
application into a menu state 1 20 that presents the user with the choices of creating a new 
log, using an existing log ,or uploading an existing log (for example, to network store 47 
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or computer 5 in the Figure 12 arrangement). If the user chooses to create a new log, state 
121 is entered in which the user is asked to specify certain details about the log (in 
particular, its name); in due course new log 100 is created and the log application 
automatically transits to state 124 in which location can be added to the log. This same 
5 state 124 is also reached when the user chooses the 'use existing log' option from the 
opening menu, the log application first entering state 122 in which the user selects from a 
list of existing logs, the log to be used; selection of the log to be used automatically moves 
the log application to state 124. 

10 When in state 124, the log application responds to an external trigger to add a location to 
the currently-selected log, by obtaining the current location of the mobile entity and 
logging it to the currently selected log together with a timestamp. The log application 
continues in state 124 with the same log selected until the user either quits the application 
or chooses to return to the menu state 120. The external trigger for adding a location can 
1 5 either be user input(e.g. by operating a hard or soft button) or a command received from 
another device. Because the log appHcation initiates location-data requests to the location 
providing means of the mobile entity, it is straightforward to arrange that the log 
application is only passed relevant location data (that is, location data it has requested) and 
therefore it will not erroneously log location data provided for other applications. 
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If the user chooses the upload option from the menu state, the log application fransits first 
to a selection state 123 in which the user selects the log to be uploaded and then to an 
upload state 1 25. In the upload state the log application oversees the fransfer of the selected 
location log. Upon completion of transfer, the log application returns to the menu state 120. 



Recording a location independently of taking a photo still has relevance to photo creation. 
For example, the situation may arise that a user would like to take a photograph of a place 
or item but has run out of film / on-camera storage, or is present at a time when it is not 
possible to take a photograph (at night, in heavy rain or mist, etc). In such cases, the user 
30 can record their location in their photo location log and subsequently retrieve from the Web 
(or other photo archive) a photograph similar to that the user wanted to take. 
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Where a camera is provided with location discovery means 29 for location stamping photos 
(see camera 90 in Figure 17), the control means 93 of the camera, when activated by user 
operation of input control 98, can be arranged to enable additional location information 
98 to be stored in memory 94 without the need to actually record image data 95; this 

5 permits the camera to log the location of desired but untaken photos. The location data that 
is recorded independently of taking a photo ('independent location data'), is preferably 
stored in sequence with location data associated with photos actually taken ('recorded- 
photo location data'); thus, for example, the independent location data can be treated as a 
normal 'image +location' data item with zero image data (see item 175). Alternatively, the 

10 independent location data can be stored in its own log separate from the recorded-photo 
location data. 



Matching Separately-Generated Image and Location Data 

In order to accommodate the separate provision of image data and location data, the album 
1 5 program 50 described above with reference to Figures 3-8 is adapted as depicted in Figure 
14. More particularly, the Load Mode is adapted to independently load the image data and 
the location data (block 141), the data loaded from the camera being handled as before but 
without the location data field being filled in on each photo meta-data record 56 whilst the 
location data is temporarily stored in a log identified as related to the batch of photos 
20 concerned. 

The Catalogue Mode is now split into two operating phases in the first of which the image 
data and location data are correlated (blocks 142 to 144), the second phase being the 
grouping and details-editing stage that formed the original Catalogue Mode. With regard to 
25 the first phase, this involves an automatic correlation process (block 142), followed by a 
user-directed correlation adjustment process (block 143); the resultant correlation of image 
and location data is then committed for storage by the user (block 144) at which time the 
location data field of each photo meta-data record is updated and the separate location log 
deleted. 



30 



In the event that the location log includes desired-but-not-taken photo location data, there 
is an additional process (see dotted block 146) between blocks 143 and 144 in which the 
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user is given the option of fetching (or initiating an automatic fetch of) photo image data 
from the mtefhet to match the location concerned. This process is depicted in Figure 1-7 
where desired image data is suppUed (arrow 172) by a specialised service 174 set up to 
provide such image data in response to requests (arrow 171). Preferably, where automatic 

5 fetching is implemented, more than one photograph will be retrieved on the basis of 
location, the user then being presented with a choice of third-party photos to add to the 
user's own photo album. As a preliminary step to fetching one or more photographs, the 
user can be presented with a detailed map 1 47 of the area around the desired-but-not-taken 
photo location 148 - the user can then specify approximately what subject/view 149 they 

10 are interested in (the location data by itself not indicating, for example, the direction in 
which the user was looking when the location was logged or whether the user was 
interested in a near field object or a far view). The user can specify the view of interest by, 
for example, clicking a target point or defining a target area on the map display. The 
information derived fi-om the user is passed with the request for retrieving relevant photos. 

15 

The user may, in fact, decide to defer fetching image data until later in which case the act 
of committing the existing correlation in block 144 also causes the creation of a photo 
meta-data record for the desired-but-not-taken photo and such ghost photos will be 
subsequently represented in the displays of Figures 6 and 7 by icons; clicking on such 
20 an icon can be arranged to initiate, at least in the Catalogue Mode, the process for fetching 
an appropriate image. 

Considering now the automatic matching process of block 142, one efficient way of doing 
this is by time-stamping digital photos in the camera and time-stamping the location data 

25 that is separately created at the same time (approximately) in a different device. Because 
different clocks are used for the two time stamps, absolute time is not reliable for matching 
the location data with the photo image data. However, the pattern of timestamps (i.e. time- 
interval related data) can be used to perform a match. This remains true even when there 
are additional entries in either the batch of photos or the location log that have no 

30 counterpart in the other collection. Figure 15 shows an example in which a timestamp 
sequence 1 50 of a batch of eight photos is to be matched against a timestamp sequencel 5 1 
of a location log with seven entries. The individual photo timestamps are represented by 
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marks 1 52 whilst the individual location timestamps are represented by marks 1 5 1 . As can 
be seen, it is a relatively easy matter to match up the two patterns of timestamps 
notwithstanding that there are two time-stamped photos 154 for which there are no 
corresponding location entries and one time-stamped location 155 for which there is no 
5 corresponding photo (this may be because the location corresponds to a desired-but-not- 
taken photo location). Appropriate pattern matching techniques for effecting the automatic 
matching of the timestamp sequences 1 50, 1 5 1 are well known to persons skilled in the art. 

The same approach could be used for matching other types of auxiliary data (and not just 
1 0 location data - for example sound clip data) with photos; again, the matching process can 
be used with any type of recording, not just photos. 

As ah-eady noted, matching can also be done on the basis of sequence number and this can 
be done even where the photos are only physical items - in this case, the location data is 
15 printed out on numbered self-adhesive labels than can be stuck to the back of the 
corresponding photos. 

Returning to matching location data and photos in the album program 150, whilst using 
sequence numbers, for example, seems an easy way to match up a set of photos with a 
20 corresponding set of location-data items, it is quite likely that there will be 
additions/omissions in one set as compared to the other. As a result the match between the 
sets will be imperfect. Mismatching may also arise where other correlation keys (that is, 
not sequence position) are used. However, it may generally be assumed that the ordering of 
entries is the same for both sets 

25 

To correct the match up, a user must intervene and manually correct en-oneous associations 
between entries in the two sets - this being the purpose of the process represented by block 
143 in Figure 14. This adjustment process can conveniently be done by generating a 
Catalogue Mode display such as shown in Figure 6 on the basis of the matching achieved 
30 after running the automatic match process of block 142 (where implemented), or else 
simply by pairing off photos with location-data items in sequence order until one of the 
sets (photos; location-data items) runs out. In the resultant display, lead lines 65 connect 
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photo thumbnails 63 with location markers 64 on the map. To correct an erroneous 
association, a user drags the map end of the relevant lead hne 65 to the correct location 
marker 64 on the map - or drags the photo end of the lead line to the correct photo (or 
simply clicks on the matching entries in turn). 

5 

To minimize the number of times this needs to be done, use is made of the consistency of 
the ordering of both sets - in particular, the associations of photos and location data for 
entries later in the orderings than a just-corrected association, are re-matched taking into 
account the corrected association. If these entries include an already corrected association, 
1 0 this latter is not disturbed. This feature is illustrated in Figures 1 6 A-D where: 

Fig. 16A - shows an initial matching of a set of photos 160 with a set of location-data 
items 161, the photos and location-data-items being paired off until the 
location-data item set is exhausted. 
Fig. 1 6B - user determines that the third location-data item 1 65 is actually associated with 
15 the fifth photo 166 and corrects the association accordingly; this resuhs in a 

re-pairing of all location-data items subsequent to the item 165 with photos 
subsequent to photo 166 as illustrated 
Fig. 16C - similarly, user determines that the seventh location-data item 167 is actually 
associated with the tenth photo 166 and corrects the association accordingly; 
20 this results in a re-pairing of all location-data items subsequent to the item 1 67 

with photos subsequent to photo 168 as illustrated. 
Fig. 16D - user now decides that the second location-data item 169 should be associated 
with the third photo 170 and corrects the association accordingly; no 
consequential downstream adjustments are made since the next association is 
25 one previously estabUshed by the user (between location data item 165 and 

photo 166). 

It will be appreciated that the match-adjustment process described above with reference to 
Figure 16 can be used to associate location data with other types of recordings. 

30 
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ANNF.y A - Mobile radio infrastructure: Lo^ atmn netermination 
This Annex forms an integral part of the specification. 

5 Communication infrastructures suitable for mobile users (in particular, though not 
exclusively, cellular radio infrastructures) have now become widely adopted. Whilst the 
primary driver has been mobile telephony, the desire to implement mobile data-based 
services over these infrastructures, has led to the rapid development of data-capable bearer 
services across such infrastructures. This has opened up the possibility of many Intemet- 

10 based services being available to mobile users. 

Byway of example, Figure 2 shows one form of known communication infrastructure for 
mobile users providing both telephony and data-bearer services. In this example, a mobile 
entity 20, provided with a radio subsystem 22 and a phone subsystem 23, communicates 

15 with the fixed infrastructure ofGSMPLMN (Public Land Mobile Network) lOtoprovide 
basic voice telephony services. In addition, the mobile entity 20 includes a data-handling 
subsystem 25 inter-working, via data interface 24, with the radio subsystem 22 for the 
transmission and reception of data over a data-capable bearer service provided by the 
PLMN; the data-capable bearer service enables the mobile entity 20 to communicate with a 

20 service' system 40 comiected to the pubUc Internet 39. The data handling subsystem 25 
supports an operating environment 26 in which applications run, the operating environment 
including an appropriate communications stack. 

More particularly, the fixed infrastructure 10 of the GSM PLMN comprises one or more 
25 Base Station Subsystems (BSS) 11 and a Network and Switching Subsystem NSS 12.Each 
BSS 11 comprises a Base Station Controller (BSC) 14 controlling multiple Base 
Transceiver Stations (BTS) 13 each associated with a respective "cell" of the radio 
network. When active, the radio subsystem 22 of the mobile entity 20 communicates via a 
radio link with the BTS 13 of the cell in which the mobile entity is currently located. As 
30 regards the NSS 12, this comprises one or more Mobile Switching Centers (MSG) 15 
together with other elements such as Visitor Ixjcation Registers 32 and Home Location 
Register 32. 
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When the mobile entity 20 is used to make a hormal telephone call, a traffic-eircuit for 
carrying digitised voice is set up through the relevant BSS 11 totheNSS 12whichisthen 
responsible for routing the call to the target phone (whether in the same PLMN or in 
5 another netv^^ork). 

With respect to data transmission to/from the mobile entity 20, in the present example 
three different data-capable bearer services are depicted though other possibilities exist. A 
first data-capable bearer service is available in the form of a Circuit Switched Data (CSD) 

1 0 service; in this case a full traffic circuit is used for carrying data and the MSG 32 routes the 
circuit to an Inter-Working Function IWF 34 the precise nature of which depends on what 
is connected to the other side of the IWF. Thus, IWF could be configured to provide direct 
access to the pubUc Internet 39 (that is, provide functionality similar to an lAP - Internet 
Access Provider lAP). Alternatively, the IWF could simply be a modem connecting to a 

1 5 PSTN; in this case, Internet access can be achieved by connection across the PSTN to a 
standard LAP. 

A second, low bandwidth, data-capable bearer service is available through use of the Short 
Message Service that passes data carried in signalling channel slots to an SMS unit which 
20 can be arranged to provide connectivity to the public Internet 39 . 

A third data-capable bearer service is provided in the form of GPRS (General Packet Radio 
Service which enables IP (or X.25) packet data to be passed from the data handling system 
of the mobile entity 20, via the data interface 24, radio subsystem 21 and relevant BSS 1 1 , 

25 to a GPRS network 17 of the PLMN 10 (and vice versa). The GPRS network 17 includes a 
SGSN (Serving GPRS Support Node) 18 interfacing BSC 14 with the network 17, and a 
GGSN (Gateway GPRS Support Node) interfacing the network 17 with an external 
network (in this example, the public Internet 39). Full details of GPRS can be found in the 
ETSI (European Telecommunications Standards Institute) GSM 03.60 specification. Using 

30 GPRS, the mobile entity 20 can exchange packet data via the BSS 1 1 and GPRS network 
17 with entities connected to the public Internet 39. 
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The data connection between the PLMN 1 0 and the Internet 39 will generally be through a 
firewall 35 with proxy and/or gateway functionality. 

Different data-capable bearer services to those described above may be provided, the 
5 described services being simply examples of what is possible. 

In Figure 2, a service system 40 is shown connected to the Internet 40, this service system 
being accessible to the OS/appHcation 26 running in the mobile entity by use of any of the 
data-capable bearer services described above. The data-capable bearer services could 
1 0 equally provide access to a service system that is within the domain of the PLMN operator 
or is connected to another public or private data network. 

With regard to the OS/appUcation software 26 running in the data handling subsystem 25 
of the mobile entity 20, this could, for example, be a WAP appUcation running on top of a 

15 WAP stack where "WAP" is the Wireless Application Protocol standard. Details of WAP 
can be found, for example, in the book "Official Wireless Application Protocol" Wireless 
Application Protocol Forum, Ltd published 1999 Wiley Computer Publishing. Where the 
OS/application software is WAP compliant, the firewall will generally also serve as a 
WAP proxy and gateway. Of course, OS/application 26 can comprise other functionality 

20 (for example, an e-mail client) instead of, or additional to, the WAP functionality. 

The mobile entity 20 may take many different forms. For example, it could be two separate 
units such as a mobile phone (providing elements 22-24) and a mobile PC (data-handling 
system 25) coupled by an appropriate link (wire-line, infirared or even short range radio 

25 system such as Bluetooth). Altematively, mobile entity 20 could be a single unit such as a 
mobile phone with WAP fimctionality. Of course, if only data transmission/reception is 
required (and not voice), the phone functionality 24 can be omitted; an example of this is a 
PDA with built-in GSM data-capable functionality whilst another example is a digital 
camera (the data-handling subsystem) also with built-in GSM data-capable fiinctionality 

30 enabling the upload of digital images fi-om the camera to a storage server. 
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As regards the service provided by the service system 40, this can be a location-aware 
service (al so known as a "location-based" or "location-dependent" ser\qce), being a service 
that takes account of the current location of the mobile entity 20. The most basic form of 
this service is the emergency location service whereby a user in trouble can press a panic 
5 button on their mobile phone to send an emergency request- for-assistance message with 
their location data appended. Another well known location-based service is the provision 
of traffic and route-guiding information to vehicle drivers based on their current position. 
A further known service is a "yellow pages" service where a user can find out about 
amenities (shops, restaurants, theatres, etc.) local to their current location. 

10 

Location-aware services all require user location as an input parameter. A number of 
methods already exist for determining the location of a mobile user as represented by an 
associated mobile equipment. In addition to location discovery systems based on GPS 
(Global Positioning System), there exist a number of other systems the most notable of 

15 which are those that rely on cellular radio infrastructures. More particularly, within a 
PLMN coverage area, it is possible to get a reasonably accurate fix on the location of a 
mobile entity by measuring timing and/or directional parameters between the mobile entity 
and multiple BTSs 13, these measurement being done either in the network or the mobile 
entity (see, for example, Intemational Application WO 99/04582 that describes various 

20 techniques for effecting location determination in the mobile and WO 99/55114 that 
describes location determination by the mobile network in response to requests made by 
location-aware applications to a mobile location center - server- of the mobile network). 

Figure 2 depicts the case of location determination being done in the network, for example, 
25 by making Timing Advance measurements for three BTSs 13 and using these 
measurements to derive location (this derivation typically being done in a unit associated 
with BSC 14). The resultant location data is passed to a location server 41 firom where it 
can be made available to authorised services. Thus, when the mobile entity 20 wishes to 
invoke a location-aware service available on service system 40, it sends a request to service 
30 system 40 via a data-capable bearer service of the PLMN 10 and the intemet 39; this 
request includes an authorisation token and the mobile entity ID (possible embedded in the 
token). The service system then uses the authorisation token to obtain the current location 
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of the mobile entity 20G from the location server 41 (the location server 41 will probably 
not be holding current location data for the mobile entity 20 and will need to request the 
appropriate BSC to determine this data before returning it to the service system 40). The 
use of an authorisation token is unnecessary if the service has been prior authorised to the 
5 location service by the mobile entity. Of course, as an altemative to having the service 
obtain location data from the location server 41 , the mobile entity could have requested its 
location from the location server and then included this information in the request to the 
location-aware service running on service system 40. 



1 0 Whilst the above description has been given with reference to a PLMN based on GSM 
technology, it will be appreciated that many other cellular radio technologies exist and can 
typically provide the same type of functionality as described for the GSM PLMN 10. 



